<html>
<head><meta charset="utf-8"><title>x.py submodule horror stories? · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html">x.py submodule horror stories?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="233222309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233222309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233222309">(Apr 05 2021 at 20:06)</a>:</h4>
<p>So, I'm about to borrow the submodule wrangling strategy that x.py/bootstrap uses for another project, since it's the most pleasant I've ever seen working with submodules get.</p>
<p>That said, it seems kind of... dangerous to do so eagerly. So I'm wondering if anybody has horror stories (<span class="user-mention" data-user-id="232545">@Joshua Nelson</span>  mentioned that <span class="user-mention" data-user-id="123586">@nagisa</span> or <span class="user-mention" data-user-id="138448">@cuviper</span> might, since they've been known to do local changes to LLVM) caused by this behavior?</p>
<p>I'm interested in failure modes, caveats, whether or not anybody's had to disable it, even just stuff like "yeah i lost a bunch of local changes once but dont really remember why", as well as general opinions on the behavior (personally it seems brilliant to me, but perhaps people who work closer to it like it less?)</p>
<p>Also, cases where its current behavior is insufficient to serve its purpose — has it ever failed to sync them for you (i know it doesnt quite work for "tracked branch"-style submodules, but these are janky anyway — mostly interested in this assuming the pattern of usage is similar to rustc).</p>
<p>(Also interested in the opposite, if you work with the submodules a lot and x.py's behavior never caused problems thats even better!)</p>



<a name="233222433"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233222433" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233222433">(Apr 05 2021 at 20:07)</a>:</h4>
<p>the horror stories with submodules that I remember predate x.py and other tooling we have in place (e.g. highfive)</p>



<a name="233222669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233222669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233222669">(Apr 05 2021 at 20:09)</a>:</h4>
<p>I think it helps that most folks <em>don't</em> want to do anything with the submodules, so it's a detail we can mostly just hide.</p>



<a name="233223200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233223200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233223200">(Apr 05 2021 at 20:14)</a>:</h4>
<p>For clarity, the algorithm that x.py uses is basically, IIUC:</p>
<ol>
<li>for submodule_path in submodules</li>
<li>run <code>git ls-tree HEAD $submodule_path</code> and parse out the hash. compare it to the output of <code>git rev-parse HEAD</code> from within the path.</li>
<li>if they're the same, skip this submodule</li>
<li>otherwise, run something along the lines of</li>
</ol>
<div class="codehilite" data-code-language="Bash"><pre><span></span><code>git submodule -q sync <span class="nv">$submodule_path</span>
git submodule update --init --recursive --progress

<span class="nb">pushd</span> <span class="nv">$submodule_path</span>
git submodule update --init --recursive --progress
git reset -q --hard
git clean -qdfx
<span class="nb">popd</span>
</code></pre></div>
<p>(although it has some logic to handle old git versions without <code>--progress</code> support)</p>



<a name="233223228"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233223228" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233223228">(Apr 05 2021 at 20:14)</a>:</h4>
<p>and yeah the case where you update the submodules would be rare for me too</p>



<a name="233223305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233223305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233223305">(Apr 05 2021 at 20:15)</a>:</h4>
<p>i figure if this caused real problems for the case where you are updating or mucking around in them, it probably wouldn't be <em>quite</em> as automatic though, or at least there would be options to disable it</p>



<a name="233223594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233223594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233223594">(Apr 05 2021 at 20:18)</a>:</h4>
<p>i'm not really sure why the second <code>submodule update</code> is needed if the first was recursive though, tbh, but i also am only sort of sure its actually there — its possible that's a mistake i introduced when reading over the logic</p>



<a name="233224042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233224042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233224042">(Apr 05 2021 at 20:23)</a>:</h4>
<p>So, for my own clarity, you havent found a need to be cautious around x.py when updating the submodules, or making local changes to them?</p>



<a name="233231089"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233231089" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233231089">(Apr 05 2021 at 21:11)</a>:</h4>
<p>I had it sync away my changes once or twice, but I learned to be careful of that</p>



<a name="233231116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233231116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233231116">(Apr 05 2021 at 21:11)</a>:</h4>
<p>and we do have an option, <code>build.submodules = false</code></p>



<a name="233247337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233247337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233247337">(Apr 05 2021 at 23:56)</a>:</h4>
<p>i see</p>



<a name="233247358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233247358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233247358">(Apr 05 2021 at 23:56)</a>:</h4>
<blockquote>
<p>I learned to be careful of that</p>
</blockquote>
<p>Can you elaborate on what you're careful of avoiding?</p>



<a name="233266164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233266164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233266164">(Apr 06 2021 at 04:46)</a>:</h4>
<p>So if you make a change to the llvm submodule, for testing or hacking something and fail to commit it, the next invocation of x.py may delete your changes.</p>



<a name="233266201"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233266201" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233266201">(Apr 06 2021 at 04:47)</a>:</h4>
<p>My solution to this problem was to just use a proper llvm checkout for the purpose.</p>



<a name="233266275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/x.py%20submodule%20horror%20stories%3F/near/233266275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/x.2Epy.20submodule.20horror.20stories.3F.html#233266275">(Apr 06 2021 at 04:48)</a>:</h4>
<p>I also hit this a couple of times but it never was major problem I feel.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>